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ABSTRACT 



Action research focused on developing approaches local 
programs can use to document outcomes of student participation in adult basic 
education (ABE) programs. Teams of teachers and administrators from three ABE 
programs examined current documentation practices, were introduced to 
approaches to documentaion, and developed documentation processes using a 
cycle of planning, implementation, and evaluation. Documentation efforts 
focused on aspects of students' lives which the program or students 
identified as areas that they hoped to change. The Virginia team established 
a process to help students identify changes they hoped to make and document 
goal and outcome achievement. The Tennessee team documented outputs and 
outcomes as part of their focus on the Equipped for the Future framework 
standard called Take Responsibility for Learning. In the Kentucky program, 
students used calendars to document activities that supported their 
children's education. Teams, programs, and students found these documentation 
efforts useful tools for instructional planning and for learner and project 
assessment. Researchers reported the following: (1) that local ABE programs 

can develop documentation processes useful for planning and assessing their 
work; such factors as limited paid time for teachers typical of adult 
education make participation in action research difficult; and action 
research is effective for professional and program development in ABE. 
(Appendixes include 30 references; participant list and sample documents.) 
(YLB) 
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EXECUTIVE SUMMARY 

The action research described in this report focused on developing approaches that 
local programs can use to document the outcomes of student participation in adult 
basic education programs. This project has implications for professional develop- 
ment as well as for outcomes documentation in adult basic education. 

Over the course of two years, three teams of teachers and administrators from 
three adult basic education programs in Tennessee, Kentucky, and Virginia, with a 
team of NCSALL researchers from the Center for Literacy Studies at the University 
of Tennessee in Knoxville serving as facilitators, addressed this issue. The teams 
examined their current documentation practices, were introduced to a variety of 
possible approaches to documentation, and developed their own documentation 
processes using a cycle of planning, implementation, and evaluation. From the 
action research, they developed new approaches to documentation and gained a new 
understanding of their work as adult educators. 

Documentation efforts focused on particular aspects of students’ lives that the 
program or the students identified as areas in which they hoped to make change. The 
Virginia team established a process that helped students identify the changes they 
hoped to make (i.e., their goals for education) and to document achievement of those 
goals and outcomes. The Tennessee team documented outputs and outcomes as part 
of their focus on the Equipped for the Future framework standard Take Respon- 
sibility for Learning. In the Kentucky program, students used calendars to document 
their activities that supported their children’s education, such as reading to their 
children and meeting with teachers. The teams, programs, and students found these 
documentation efforts useful tools for instructional planning and for learner and 
project assessment. The programs have continued to use these documentation tools 
and to develop new ones. 

Although the main purpose of this research was to contribute to under- 
standing of a systemic issue — how to document the outcomes of participation in 
adult basic education — it also contributed to the participants’ professional, program, 
and personal development. For the participating teams, this action research project 
has led to increased understanding of how programs might identify and document 
student outcomes in ways that meet local program needs and how that documen- 
tation can support program improvement. Team members learned from working in 
groups and having an opportunity to share experiences with other teachers in their 
own teams and from other teams. The two aspects of this project that participants 
seemed to particularly value were the focus on students and the opportunity to reflect 
on the goals of their work. 
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When the teams were asked to identify challenges, some named conceptual 
issues, such as “the idea of documenting changes in learner lives versus test scores,” 
but most named practical challenges common to the field of adult basic education. 
These included student turnover; limited time for teachers to design, collect, and 
implement documentation; and difficulty in finding ways to share results with 
teachers in other programs. 

The facilitators extensively reviewed project data. The findings from this 
review with direct implications for practice and policy in adult basic education are: 

• Local adult basic education programs can develop documentation processes 
useful in planning and assessing their work. However, more work is needed on 
the local, state, and national level before these locally developed documentation 
processes can be used for the performance accountability systems the Workforce 
Investment Act requires. 

• Action research is an effective approach for professional and program 
development in adult basic education, but if it is to be widely used, factors such 
as the limited paid time typical of adult education in most states must be 
addressed. 

On the basis of these findings, the researchers make the following recom- 
mendations to the field of adult basic education about outcomes documentation and 
using action research as a tool for professional development, program improvement, 
and performance accountability: 

• Those responsible for professional development in adult education should use 
action research more extensively. To improve practice, teachers need to be paid 
for the time they spend in action research, action research needs to be accepted 
as professional development by the state system, and facilitation support needs 
to be available. 

• Local and state adult basic education (ABE) administrators should encourage the 
use of action research approaches to improve program quality. Systematic 
processes of reflection to identify areas that need improvement, combined with 
ongoing action and evaluation, help keep a program focused on continuous 
improvement. 

• States should build consensus about the goals underlying their performance 
accountability systems, using such participatory processes as action research. 
Action research as professional development should include this local definition 
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of goals as part of consensus building because this focus and measuring goal 
achievement seem to build a program’s capacity to implement performance 
accountability systems. 

• On the state and federal levels, resources should be committed to designing 
outcome measurement and reporting systems flexible enough to include a 
variety of goals and rigorous enough to measure performance. 
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CHAPTER 1: INTRODUCTION 

Teacher: They ’re working on their businesses in the classroom. So they were using 
skills to try to buy this computer— pretend, you know. But, still, if I had a folder and 
I had a check-off list that said, “They negotiated, they listened actively, they spoke so 
others could understand, they were respectful to the computer teacher, ” then I could 
actually check that off. And I could have that in the folder. And then I could go back 
. . . and I could say, Okay, this helped me see that they did this well. I can see that 
being a way into it, a way to do it. 

Researcher: It just struck me that one of the things we 're trying to do here is capture 
little moments, when the process is very dynamic. We ’re trying to figure out how you 
give evidence for moments that tell you that you ’ve moved. It ’s never going to be the 
whole picture. 

Administrator: Seeing what we do in documentation really ties together our 
students ’progress. 



Over the course of two years, teams of teachers and administrators from three adult 
basic education programs addressed how local programs might document the 
outcomes in students’ lives of their participation in adult basic education programs. 
The teams examined their current documentation practices; were introduced to a 
variety of possible approaches to documentation; and developed their own processes 
using a cycle of planning, implementation, and evaluation. From the action research, 
they developed new approaches to documentation and a new understanding of their 
work. This report describes the work of the project and what can be learned from 
this action research. 

Two major national initiatives, the National Reporting System (NRS) and 
Equipped for the Future (EFF), influenced this action research project. In 1998, 
Congress passed the Workforce Investment Act (WLA) mandating a performance 
accountability system for federally funded adult basic education programs in the 
United States (P.L. 105-220). States are required to set levels of performance for 
three core indicators: 

• Demonstrated improvements in literacy skill levels in reading, writing, and 
speaking the English language; numeracy; problem solving; English language 
acquisition; and other literacy skills 
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• Placement in, retention in, or completion of postsecondary education, 
unsubsidized employment, or career advancement 

• Receipt of a secondary school diploma or its recognized equivalent 

The NRS establishes the measures that states may use in their reports on 
the WIA core indicators. It also provides optional secondary measures that a state 
may report (for instance, registering to vote, increased involvement in children’s 
education, or leaving public assistance), but these are not included in state per- 
formance assessments. The NRS was being implemented as the action research 
teams carried out their work. 

The National Institute for Literacy’s Equipped for the Future (EFF) 
standards-based system reform initiative has conducted a multi-year, field-based 
research process to determine what adults need to know and be able to do in their 
roles as workers, family members, and citizens (Stein, 2000). When completed, EFF 
will provide a common framework for defining, tracking, and reporting results to 
policymakers as well as to students and their local programs. The EFF framework 
consists of: 

• Four Purposes for Learning, defined originally by adult learners and validated by 
a wide range of adults. These purposes are access to information so adults can 
orient themselves in the world; voice, or the ability to express ideas and opinions 
with confidence; independent action, or the ability to solve problems and make 
decisions independently; and a bridge to the future, or learning how to leam, to 
keep up with a changing world. 

• Three role maps that define activities critical to carrying out the roles of worker, 
citizen, and family member, such as Become and Stay Informed to be an effective 
citizen, Promote Family Members ’ Growth and Development to be an effective 
parent or family member, and Work Within the Big Picture to be an effective 
worker. 

• Thirteen activities common across these three roles, such as Manage Resources, 
Guide and Support Others, Create and Pursue Vision and Goals, and Keep Pace 
with Change. 



2 




12 



NCSALL Reports #20 



March 2002 



• Sixteen Content Standards that provide specific and measurable statements of 
what adults need to know and be able to do, clustered in four categories: 
communication, interpersonal, decision-making, and lifelong learning skills. 
Each Standard describes components of skills typically taught in adult basic 
education, such as Read with Understanding and Resolve Conflict and Negotiate. 

EFF has been developing an assessment framework to define performance 
levels and identify progress measures for the skills identified as their standards for 
what adults need to know and do to carry out the key activities in their lives. This 
work began in 2001 and will be completed in 2003. 

In the long term, EFF is addressing what adults should know and be able to 
do and developing new approaches to assessing learner progress. The NRS has 
established how to measure a narrow range of skill gains and limited outcomes, at 
least in the short term. However, questions remain about documenting outcomes of 
adult literacy education in students’ lives and about how local programs might 
document outcomes in ways that meet student and practitioner needs. EFF expects 
to eventually provide a way for programs to show “results that matter” for all 
stakeholders, and the NRS establishes ways to document particular outcomes that 
concern policymakers. In our study, we focused on how local programs might 
document outcomes in ways immediately useful to students, teachers, and programs. 
We determined that an action research project might be an effective way to explore 
this issue. 

This project was a part of the work that the Center for Literacy Studies (CLS) 
at the University of Tennessee in Knoxville conducted as a partner in the National 
Center for the Study of Adult Learning and Literacy (NCSALL). The CLS work for 
NCSALL has addressed how to assess the impact of literacy learning in ways that 
serve policymakers concerned about the results of their investment, practitioners 
concerned about the efficacy of their work, and adult students concerned about how 
their efforts to learn will benefit them in the rest of their lives (Merrifield, 1998). 

This is particularly relevant in a time of increased emphasis on performance 
accountability. 

The CLS and NCSALL efforts to address assessment of the impact 
of literacy learning have included: 

• A policy paper examining performance accountability in adult education 
(Merrifield, 1998) 
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• A paper reviewing previous outcomes studies in adult literacy education 
(Beder, 1999) 

• Two studies examining how learners assess the changes in their lives resulting 
from participation in adult literacy programs (Bingman & Ebert, 2000; Bingman, 
Ebert, & Smith, 2000) 

• A policy paper based on the findings (Bingman, 2000) 

The remainder of this report includes a methodology chapter briefly dis- 
cussing action research and giving an overview of the project processes, a chapter 
describing the first and second project stages, a chapter describing the results of the 
teams’ work in Stage 2, a chapter discussing the CLS researchers’ findings, and a 
chapter presenting conclusions and recommendations. This report is based on our 
field notes, project team reports, and project artifacts the facilitators and program 
teams produced. 
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CHAPTER 2: METHODOLOGY 
Action Research 

Action research is an approach to research and development grounded in practice. In 
a collection of articles edited for an adult education audience, Kuhne and Quigley 
(1997) described action research as “a form of inductive, practical research that 
focuses on gaining a better understanding of a practice problem or achieving a real 
change or improvement in the practice context” (p. 23). They described a repeated 
cycle of six steps in three phases: 

Planning Phase 

1. Pose problem 

2. Define project and determine intervention 

3. Determine measures of data collection 

Action Phase 

4. Implement action and observe results 

Reflection Phase 

5. Evaluate the results 

6. Reflect on the project, possibly posing another problem 

In the same volume, Creating Practical Knowledge Through Action 
Research: Posing Problems, Solving Problems, and Improving Daily Practice 
(Quigley & Kuhne, 1997), Quigley discussed the development of action research, 
from the work of Kurt Lewin to the reflective practice of Argyris and Schon to the 
more radical participatory action research advocated by Kemmis and McTaggart. 
Quigley situated action research in the practice of adult education in the 
“institutional and staff development context” (p. 16) and as an instrument of 
empowerment and social change. 

In describing participatory action research, McTaggart (1997) emphasized 
that action research involves “the intensive study of a situation and the production of 
knowledge in some form or another, including important ideas like informed 
practice” (p. 27). He discussed Lewin’ s action research cycle as “a spiral of steps, 
each of which is composed of planning, acting, observing, and evaluating the results 
of the action” (p. 27) and described action research as a joint project of academics 
and “workers” (p. 31). The guiding principles of participatory action research 
McTaggart outlined include: 
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• Concern on the part of academics and practitioners (workers) with under- 
standing and improving both their own individual and their organizational 
practice 

• Studying and changing the discourse and practice, including distribution of 
power of the organizations involved 

• Getting started quickly and starting small, with a spiral of planning, acting, 
observing, and reflecting 

• Using the project to both change practice and produce knowledge 

• Beginning with the subjective experiences of the participants 

McTaggart described projects that began as small cycles of planning, action, 
observation, and reflection but built over time to include multiple cycles and 
extensive documentation of the changes in activities, discourse, social relationships, 
and forms of organization and development of expertise (p. 39). These iterative 
processes culminate in research in which evidence and critical reflection combine in 
new critiques, knowledge, and theory, as well as in changed practice. 

The Documenting Outcomes Project 

Although this project included some of the elements of participatory action research 
that McTaggart described, our work has been more limited — by both time and 
resources. With the participating teams, we developed processes that enabled both 
academics and practitioners to build understanding of their work and the contexts in 
which they work. The teams started small and used expanding, iterative processes of 
planning, action, and reflection. The project began with participants’ experiences, 
has changed practice, and produced knowledge about that practice. The teams did 
not — except incidentally — study practice discourse or power distribution. Although 
the project did not include the level of documentation or length of process to build 
new critiques, knowledge, and theory, it may contribute to a critique of limited, 
highly bounded ways of measuring outcomes. 

The methodology in action research is not necessarily predetermined. Its 
description, therefore, is messier and more complex than that found in more tradi- 
tional research. Our action research design included four not always linear steps: 
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• Understanding the current situation in local programs, what was and what was 
not documented, and why 

• Examining various frameworks that integrate documentation into program work 

• Developing and implementing new approaches 

• Reflecting on the results 

The project’s work occurred in two stages, each including cycles of planning, 
action, and reflection. The first stage, in 1998, involved one program and focused on 
clarifying the project’s issues and processes. The second stage, in 1999-2000, 
involved three programs and was more structured. Both stages contributed to greater 
understanding of the issues and led to new approaches to documenting outcomes. A 
third stage is ongoing, as the three programs continue to build on their work in this 
project. Each stage included: 

• Understanding the situation and clarifying the questions 

• Examining frameworks, particularly EFF, that could structure action to document 
outcomes 

• Developing and implementing new approaches 

• Reflecting on results 

CLS staff serving as facilitators and documenters and teams of practitioners 
from three adult education programs carried out the project work. For the names and 
positions of the staff and team members, see Appendix 1 . 

In the first stage, the methodology focused on understanding the situation and 
posing the problem. The facilitators worked with one program to define terms and 
clarify the meaning of “outcomes.” At the same time, the program explored the EFF 
framework and the ways it might contribute to documenting outcomes. In the 
second stage, the facilitators used varied activities to help the teams understand their 
situation in terms of documenting outcomes. In both stages, the program teams did 
the primary work in developing and implementing new approaches. The following 
table gives an overview of the project and the three teams’ work. 
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Table 1: Overview of the Documentine Outcomes Project 





Tennessee: Knox County 
Adult Literacy Program 


Virginia: Mount Rogers Regional 
Adult Education Program 


Kentucky: Knott County 
Adult Learning Center 


Stage 1 


Fall 

1997 


• Became EFF field site 

• Began to meet with CLS 
facilitators to plan the action 
research 






Winter 

1998 


• Team attends EFF Institute 

• Team experiments with using 
EFF, e.g., students rewrite EFF 
poster 






Spring 

1998 


• Inputs-to-Impacts model 
developed 






Fall 

1998 


• Teachers begin implementing 
outcomes documentation 
processes 

• Start second round of EFF 
field research 






Stage 2 


Winter 

1999 


♦ Continuation of documenting 
outcomes work 

• Completion of documentation 
matrix 


• Join action research project 

• Reflect on student outcomes 

• Complete documentation 
matrix 

• Implement and evaluate 
learning activities 

• Review EFF 

• Revise Inputs to Impacts 
model 


• Join action research project 

• Reflect on student outcomes 

• Complete documentation 
matrix 

• Explore learning activities 

• EFF introduction 

• Revise Inputs to Impacts 
model 


Spring 

1999 


• Regional meeting 

• EFF Debriefing Institute 


• Regional meeting 

• Revision of state reporting 
forms 


• Regional meeting 

• Work on Parent Role 

• Map with students 


Summer 

1999 


• Decision to focus on 

documenting EFF standard 
Take Responsibility for 
Learning (TRL) 


• Developed and revised 
DoSetMet form 


• Summer Reading Project with 
calendar documentation 


Fall 

1999 


• Design and implement TRL 
process 

• Regional meeting project 


• Pilot DoSetMet with lead 
teachers and Washington 
County 

• Evaluation meeting 

• Regional meeting 


• Implement expanded calendar 

• Regional meeting 


Winter 

2000 


• Continued to use 

documentation processes 


♦ Continued to use 

documentation processes 


• Continued to use 

documentation processes 


Spring 

2000 


• Final reports 


• Final reports 


• Final reports 
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The Participating Teams 

In the first stage of the project, the facilitators worked with a team from the Knox 
County Adult Literacy Program (KCALP). We chose KCALP as a research site 
because of its interest in EFF, history of work with the CLS, and interest in 
developing a system of outcomes documentation as part of a continuous improve- 
ment process. KCALP served Level 1 students (testing below sixth-grade level in 
reading or math) as part of the county school system’s adult basic education 
program. Located in an urban area in downtown Knoxville, Tennessee, KCALP 
works closely with the nonprofit Friends of Literacy to provide services. Both day 
and evening classes were offered at the main center, and Friends of Literacy operated 
family literacy classes in a Knoxville apartment development. At both locations, day 
classes met for five hours on each of four days per week, and evening classes were 
held six hours a week. The two programs had eight full-time teachers and 59 active 
volunteers working with 220 students. 

In consultation with state adult basic education staff in Kentucky and 
Virginia, two teams were added in the second stage. State staff were given a project 
description and asked to recommend programs for the project. (Artifact 1 is a flyer 
sent to state and program staff.) In both Virginia and Kentucky, the programs the 
states recommended were interested in participating. CLS staff met with program 
administrators and developed work agreements. KCALP also continued to 
participate in Stage 2, but on a slightly different schedule. At all three sites, team 
members were paid an honorarium for the extra time the project required. 

The Knott County Adult Learning Center is located in the coal-mining region 
of the eastern Kentucky mountains. In 1999, the program enrolled 70 students in 
adult basic education (ABE) classes and 23 in literacy classes. Classes were offered 
six hours a day in an adult learning center in the county seat. The majority of the 
program’s students were young single mothers. All five staff members (three GED 
instructors and two literacy instructors) were on the action research team. Two 
were professional teachers and administered the program; the others were para- 
professionals. 

The Mount Rogers Regional Adult Education Program is located in rural 
southwestern Virginia and serves five counties and two small towns. This program 
serves 1,200-1,500 students per year in English as a second language (ESL), ABE, 
and workplace classes. The program also works with community colleges to help 
prepare students without high school diplomas and provides adult basic education in 
several correctional facilities. The program has about 40 part-time teachers and two 
full-time administrators. Class schedules vary, but most classes meet for a few hours 
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once or twice a week in a variety of sites. The Mount Rogers action research team 
included three instructors from one county. Their classes included a weekly GED 
class held in a local library, classes in a public housing development, a class at the 
county vocational school, community classes held in a high school, and classes in 
the county jail. The lead teacher from that county also participated in the project’s 
final stages. 

Artifact 1 ; One-Page Description of Action Research Project Used in 
Program Recruitment 



Documenting Outcomes for Learners and Their Communities: 

Developing Performance Accountability at the Local Level 
A Center for Literacy Studies NCSALL Project 

What: Action research projects to develop ways to document the outcomes of participation in adult basic 
education programs on the quality of life of adult learners and their communities. 

When: January 1999 to December 1999 

Who: Teams of 3-5 teachers and 2-3 administrators from three programs in Tennessee, Virginia, and Kentucky, 
working with staff from the Center for Literacy Studies. 

Why: To contribute to the development of knowledge about performance accountability systems by exploring 
ways that local programs can document outcomes. 

How: 

• Teams will develop processes for documenting outcomes of participation in adult education in learners’ 
lives and communities and will implement these documentation processes in their program on a trial basis. 

• The teams will explore using the Equipped for the Future (EFF) standards as a framework for their 
documentation and will consider quality of life indicators and measures used in fields such as community 
development. 

• The teams will consider connections to state and federal performance accountability systems as they develop 
their processes. 

Project Activities 

• Teams will identify possible/probable outcomes for learners’ lives. 

• Teams will explore connections with EFF framework. 

• Teams will develop methods to try out ways to document outcomes. This will not be a comprehensive 
system but will focus on a few outcome areas. 

• Teams will evaluate documentation process tried by team members. 

• Teams will implement one or more effective processes programwide. 

• Teams will identify connections with state systems. 
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Summary 

To summarize the project methodology: 

• The project was framed as action research, defined by Kuhne and Quigley (1997) 
as “inductive, practical research that focuses on gaining a better understanding of 
a practice problem” (p. 23). The need to document outcomes of literacy edu- 
cation that are possible and useful at the program level was the problem in this 
instance. 

• Three adult education programs teams participated in the project. A team of CLS 
researchers facilitated their work. 

• The research was conducted in two stages, the first with one program team, the 
second with two additional teams. 

• The research process began with a series of activities that enabled teams to 
examine their current practice and consider how outcomes documentation fit 
with their program needs. 

• The facilitators introduced possible approaches, including EFF, that might prove 
useful in this effort. 

• Each team built on these activities through a cycle of planning, implementation, 
and evaluation to develop their own documentation processes. 
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CHAPTER 3: IDENTIFYING THE QUESTION, PLANNING THE WORK 

Both Stage 1 and Stage 2 of this action research project included a planning cycle. 

In Stage 1, planning and building understanding took the greater part of the year. In 
Stage 2, the planning processes were more systematic, and the teams spent more time 
developing and implementing their documentation processes. This chapter describes 
the activities used to plan and build understanding in both stages. The CLS 
facilitators designed and led these activities and provided feedback and support to 
the teams. 

Stage 1: The Tennessee Team Explores Documenting Outcomes 

KCALP carried out the first stage of the project. The Documenting Outcomes action 
research project was one of KCALP’s three major projects in 1998, and it was 
integrated with and supported the other two projects. In 1997, KCALP had moved 
from an ambitious strategic planning process to the even more ambitious project of 
continuous improvement as structured by the Malcolm Baldridge Educational 
Criteria for Performance Excellence, which focus on improvements to such 
organizational aspects as leadership and work systems. At the same time, KCALP 
was a partner in the EFF field development work. KCALP has been very deliberate 
in choosing to be involved in projects that support its long-term strategic goals and 
chose to continue their EFF collaboration and participate in the action research 
project because these efforts would advance its goals. (For more about KCALP’s 
program improvement efforts, see Cody, Ford, & Haywood, 1998.) 

KCALP’s participation in the action research project started as a team of four 
teachers and two administrators (who also taught) began work on the EFF standards 
field development project. For this, the team developed a program profile, studied 
EFF, attended two training institutes, and conducted in-class research on the EFF 
standards. Teachers used selected draft EFF content standards to plan and teach 
lessons and document student performance. They reported on this process and 
recommended changes in the standards. 

While KCALP teachers experimented with using the EFF framework, 

KCALP team leaders met regularly with the NCSALL facilitators from the CLS and 
identified and clarified issues to address in the Documenting Outcomes action 
research. CLS staff reviewed the literature on quality of life measurement, which 
informed the discussions. The action research team also reviewed logic models that 
the United Way of America (1996) and others (Flora, Flora, & Wade, 1996) used to 
move beyond a simple input-to-output evaluation model, in which only such 
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immediate activities as number of clients served are considered, to evaluations that 
capture more meaningful changes or outcomes. 

Based on the logic models they reviewed, the facilitators and the KCALP 
team developed the Inputs-to-Impacts model to clarify the various aspects of adult 
education program processes. The model lists student and program factors 
separately and defines input as the factor available for performance (i.e., what the 
student or program brought to the processes of the program). The processes include 
the educational and organizational processes a program implements. Outputs are 
defined as the immediate results of these processes, whereas outcomes are the 
changes that occur in students’ lives through participation in the program or the 
long-term results of program improvement. Impacts are the changes in the 
community resulting from changes in students and programs. This “Inputs-to- 
Impacts” model became an important tool in the action research process. The action 
research teams found the model useful in both analyzing their broad program 
structure and examining particular activities (e.g., disentangling program outcomes 
from individual student outcomes). The version of the Inputs-to-Impacts model in 
Artifact 2 is a revision that includes items from all three action research teams. 
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Artifact 2: Logic Model Framework Developed by Programs 

Inputs-to-Impacts Model: 

A tool for analyzing performance factors in ABE programs 





Student 


Program 


Inputs: 

Factors available for 
performance 


• Previous educational experiences 

• Life experiences 

• Goals 

• Abilities 

• Commitment to learning needs 

• Temperament (e.g., shyness) 

• Challenges (e.g., learning disability, 
childcare needs, transportation) 


• Building, equipment 

• Staff 

• Curriculum 

• Materials 

• Program goals 

• Volunteers 

• Technology 


Processes: 
Educational and 
organizational 
processes contributing 
to performance 


• Intake interview 

• Orientation 

• Reading, writing, math activities 

• Social interactions 

• School governance activities 

• Testing and assessments 

• Discussion/analysis 

• Cultural expressions 

• Computer use 


• Planning 

• Record keeping 

• Assessment 

• Scheduling 

• Instruction 

• Staff development 

• Advice, guidance, support 

• Referrals to human services 


Outputs: 

Immediate results of 
services provided 


• Test scores 

• Journals 

• More comfort in class 

• GED 

• Resume 

• Certificates 

• Documentation of improved 
performances 


• Number of classes offered 

• Number of hours of instruction 

• Number of students 

• Number of staff development 
activities 

• Records kept 


Outcomes: 

Longer-term results of 
education for 
individuals and 
programs 


• New reading, writing, math 
practices 

• Changed self-concept 

• Opened a checking/savings account 

• Computer skills 

• New goals 

• Changed/new skills 

• Driver’s license/commercial 
driver’s license 

• Workforce skills 

• Citizenship 

• Job or job promotion 


• Aggregation of student outcomes: 

test scores 
GED 

student goals met 

• Teacher changes 

• Improvement in program quality 

• Changes in program philosophy 


Impacts: 

Changes in community 
brought about by 
changes in learners’ 
lives 


• Children more involved in school 

• Increased use of public resources 

• More activity in civic life 

• Pressure for improved neighborhood 

• Better educated/developed workforce 



This draft was produced by the staff and action research teams of the CLS NCSALL Documenting Outcomes 
Project, 1999. 
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As the KCALP teachers used the EFF framework, they explored ways to 
integrate the broad EFF skill standards with the basic academic skills they continued 
to teach. They began to focus more on helping students identify life goals and 
connecting instruction and learning to those goals. EFF’s role maps and focus on 
learners’ purposes helped make those connections. The KCALP team — teachers and 
administrators — identified the need for assessment measures that went beyond the 
standardized tests they used. They needed ways to assess learning gains as well as 
outcomes of goal achievement. Although students often told them about outcomes, 
they had no systematic way of recording those oral reports. They discussed evidence 
of performance of EFF skill standards and what would serve as evidence of goal 
achievement. In fall 1998, the KCALP team began to focus on ways to document 
outcomes. Rather than collecting evidence on a broad range of possible outcomes, 
they decided to focus on learner goals and ways to document achievement of 
particular goals and the resulting outcomes. 

Beginning to Document Outcomes 

Each of the four KCALP teachers used a different approach to document outcomes, 
but all based approaches on learner goals. One teacher worked closely with two 
students who wanted to open their own business. She met with them twice a month 
to discuss their progress and used taped interviews as a documentation method. The 
students started a cleaning business and also identified changes or outcomes. These 
included increased self-esteem, discovery of capabilities needed to start at business, 
greater comfort speaking in front of others, and an ability to solve problems. A 
second teacher documented students’ use of math — specifically measurement 
skills — as they painted and decorated their new classroom. She classified the work 
completed in class — learning to compute area and perimeter, measuring the room, 
and drawing scale models — as outputs, whereas the newly decorated classroom and 
student reports of using measurement skills at home were classified as outcomes. 

The outputs and outcomes were documented with artifacts or reports in student 
portfolios. A third teacher tried using student journals to document use of a math 
skill (estimation) as an outcome in learners’ lives. A fourth teacher kept a collection 
of products students had produced in the computer lab to meet needs in their 
everyday lives, an invitation or a flyer for a home business, for example. 

While the teachers developed documentation processes for students in their 
classes, the team’s two administrators focused on student outcomes resulting from 
participation in a student leadership team. The administrators also began 
systematically reviewing test scores with students and discussing errors, helping 
students identify better learning strategies. The KCALP team summarized their 
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work in the first stage of this action research on documenting outcomes in an 
adaptation of the Inputs-to-Impacts model, found in Appendix 2. 

When the KCALP team reviewed their work in Stage 1, they concluded they 
had accomplished a great deal in terms of understanding ways to structure their 
instruction and tie it to student goals. The team members were using EFF to both 
frame instruction and support their continuous improvement work, building student 
leadership and integrating a month-long “Learning Skills Class” orientation into the 
broader program. But the teachers’ efforts to document learning outcomes in 
students’ lives, particularly outcomes outside the classroom, had not proceeded as 
they had hoped. Teachers were confused about the difference between outputs and 
outcomes. The processes they had developed were time consuming and did not 
enable teachers to readily document and report outcomes to other stakeholders. In 
the second stage of the project, KCALP tried a new approach more directly grounded 
in EFF. This is described in Chapter 4. 

Stage 2: Virginia and Kentucky Join in a More Structured Process 

The second stage of this action research project built on the work and experiences of 
KCALP in Stage 1 and broadened the project to include the Virginia and Kentucky 
teams. Although the EFF framework continued to inform the project, the two new 
teams were not directly involved in EFF development work. Both new teams were 
from rural programs and added the perspectives of two different states. 

During Stage 1 of this project, the KCALP team members largely developed 
their action research work themselves, with guidance from CLS facilitators. During 
the Stage 2 year (1999), CLS facilitators structured each team’s initial work. CLS 
staff held about a dozen meetings with each team, the earliest including specific 
activities to introduce the project and begin developing documentation processes. 

The CLS facilitators collected data on these activities, including agenda, minutes, 
and field notes from each meeting. A sample facilitator’s agenda for a team meeting 
can be found in Appendix 3. 

Activities common to all three sites (with some variation for KCALP) 
included: 

• An initial activity to define the issue 

• Completion of a documentation matrix of current documentation processes 

• Exploration of instructional activities to document learner change 
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• An introduction to EFF 

• A review of the Inputs-to-Impacts model 

These activities, as well as activities at two regional meetings all three teams 
attended, provided opportunities and structures to explore the theory and practice of 
documenting outcomes. 

After the series of initial meetings, the program teams began to develop their 
own documentation processes. The later meetings were times to report and reflect 
on and sometimes revise the teams’ work. Teams reported on their ongoing work at 
meetings with the CLS facilitators. Each team’s work developing outcomes 
documentation processes particular to their programs is described in Chapter 4. 

Defining the Issue: What Are the Outcomes We Care About? 

The initial meeting with each team began with a process to examine what was meant 
by “outcomes” and to begin to determine the outcomes documented in each program 
(the KCALP process was somewhat different because of its earlier work). Each 
team member was asked to think of two particular students and write about changes 
in these students’ lives that may have resulted from their participation in adult 
education programs. The changes or outcomes were listed on newsprint and 
discussed in terms of types of outcomes and reasons to document. We noted that the 
outcomes the team members named were often not those the program documented. 
Artifact 3 is drawn from newsprint created in the initial meeting of the Virginia team. 
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Artifact 3. Notes from First Virginia Team Meeting, 2/1/99 



Outcomes listed for an ESOL student: 

• Learned English 

• Learned to follow directions in English 

• Uses map 

• Much stronger confidence 

• Uses English to interact with other students 

• Built friendships 

• Uses phone to make inquiries 

• Uses Internet to find information 

• Uses computer for Internet and educational activities 

Outcomes listed for a student in an ABE class held in a public housing project: 

• Uses Internet to locate parenting and health information 

• Able to critically read Internet material 

• Uses Internet to follow interests 

• Read a book 

• Opened checking account 

• Got a job 

• Read school reports 

• Tried new parenting skills 

• Improved use of phone book 

Outcomes for a student in a jail class: 

• Reading level improved 

• Reading Bible 

• Passed all but math chapter on the GED practice test 

• Doesn’t curse anymore 

• Plans to drive truck 



The Documentation Matrix 

Developing a matrix helped the teams focus on the documentation processes that 
they and their programs already used. The facilitators asked the teams to bring all 
their documentation forms to the meeting in which they developed a matrix. The 
meeting room tables were covered with tests, folders, forms, and printouts. Using 
newsprint or a whiteboard, the teams listed their program’s various forms of 
documentation and answered the following questions about each piece: Who does it? 
For whom? How often? How is the information used? What are the key items 
reported? Artifact 4 is the Kentucky team’s matrix. 
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After completing this process, the teams examined program goals: those 
established by the state, those determined by the program, and their individual goals 
for their work. The teams then looked at the matrix they had created and noted that 
accomplishment of many of these goals, particularly their own goals for their work, 
were not being documented. For example, in Kentucky, both the program and state 
goals were primarily focused on inputs — that 6 percent of the population would have 
its educational needs served — whereas the teachers’ goals included “ensuring that 
every student feels like they have experienced success in the program,” “fostering 
the process of students gaining self-esteem and taking on leadership roles,” and 
“helping students see the importance of personal responsibility.” The documentation 
identified in the matrix process measured attendance and achievement of a variety 
of state-identified objectives on a checklist, for example, “earn a GED,” “learn 
wellness/health.” The program did not have a way to document the other outcomes 
they believed were important to students’ lives. 
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Using Instructional Activities to Document Outcomes 

To begin exploring approaches to documenting outcomes, the facilitators suggested 
several learning activities that involved students in reflecting on the outcomes of the 
adult education experience in their lives. Each team member was asked to try one of 
these and record the results. Suggested activities included: 

• Students write a sentence, paragraph, or essay (depending on skill level) starting 
with the stem sentence, “Since I started adult ed classes, my life has changed . . .” 

• Teacher leads a story circle in which each person tells a story of a change in his or 
her life. Teacher records on a flipchart. 

• Student identifies a goal (e.g., helping children with schoolwork) and keeps a 
calendar to record evidence of meeting that goal. 

• Student keeps a portfolio of items that show outcomes of changes in their lives, 
(e.g., a canceled check copy if they opened a checking account). Student and 
teacher decide what is evidence. 

• Student and teacher identify a short-term goal and ways the student would know 
he or she is meeting that goal (the evidence). Together, they make a checklist, 
and, as the student does one of these things, it is dated on the list. For example, if 
a student’s long-term goal is getting a GED but he or she needs numerous math 
skills to get there, the checklist might be the various math skills. 

Team members in Kentucky and Virginia tried these activities and recorded 
the results. Some team members used an activity once, and others used the activity 
in an ongoing process. For the most part, the teams did not find these activities 
effective as outcomes documentation. With some of the stem sentences, reported 
outcomes were very general, and students did not reflect on or provide evidence of 
outcomes. One teacher used a four-page set of stem sentences requiring specific 
outcomes and evidence, but she reported that the process took too long and students 
resisted taking the time. Using calendars and lists helped focus on particular 
learning objectives but not on outcomes in people’s lives. Three teachers tried a 
“story circle.” One interpreted this as a group discussion to address a particular 
topic — emergency phone numbers — and the outcomes she identified were her own 
assessments of student change (e.g., increased self-confidence). She did not 
document student responses. The other two teachers focused discussions on changes 
in students’ lives, but only one framed the discussion in terms of how being part of 
the class “had helped them [students] individually.” An excerpt from this teacher’s 
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log is reproduced here. This teacher reported that the activity had given her a 
structured, helpful way to talk to students about their lives, and both she and her 
students were excited about the changes they recognized. However, except in her 
log of the discussion, the outcomes were not recorded or integrated into a broader 
system of outcomes documentation. 

Artifact 5: Notes from Teacher’s Log, 2/26/99 



What did 1 do? 

I organized a group discussion with three students. The group consisted of two students from the School-to- 
Work/GED program and an ESL student. I began our discussion with telling the group that, as instructors, we 
were able to document their academic progress but that I was also interested in their personal progress, how being 
a part of the GED program had helped them individually. We began our discussion with: 

“What can you do now that you couldn’t do before?” 

1. I can read! 

9 2. / can use a ruler, and this has helped me with my vocational class and projects at home. 

3. I am more committed to meeting the goals I have set. Before entering the program, I was not very 
responsible, and I didn 7 know how to work towards something, but now I do. I have also learned study 
skills that have carried over into my other (vocational) class. 

4. / am not afraid to talk to other people or to ask a question. 

5. I can communicate in English. I can read and understand English. 

“Have you noticed any changes at home with yourself or family members?” 

1. lam reading books and the local newspaper. 

2. I am able to communicate with my mother. / used to always be negative about myself and everything else, 
and that caused a lot of problems between my mother and /. I try to be positive, and that has helped the two 
of us get a long better. I can express myself better. 

3. My father is learning English now. I help him a lot. Before, we only talked to each other in Chinese, but 
now we talk some in English. We now want my mother to learn. 

4. / study English on my own every day at home. I read the newspaper, history books, and other books. 

5. Sometimes we get things in the mail, and my father does not understand it, but I can now read it and explain 
it to him. 

“What about work?” 

1. I am better organized and work harder than before. I want to accomplish more than what is expected. 

2. I am more committed and dependable. 

3. I had never used a computer before entering this program. I can now use a computer in my vocational 
class. I can also use a copy machine and a fax machine. 

4. I am able to understand my customers better and can help my father, mother, and uncle understand them. 

What did I learn? 

The students were excited to share their thoughts and seemed to be even more excited about the changes they had 
noticed in themselves. I was also excited about our group discussion. It proved to me that we don’t just affect 
our students academically, but we have a great impact in all aspects of their life. I really enjoyed being with them 
and talking with them. 
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Team members reported that these instructional activities were useful ways to 
know students better and helped students reflect on goals and accomplishments. 
Trying the activities also seemed to help some of the team members experiment with 
new instructional approaches. However, the team members did not believe these 
activities would meet the program’s documentation needs. As the teams developed 
their own documentation processes, they integrated some elements of the activities 
but developed more structured approaches. 

EFF 

A review of EFF was included as an action research activity because it informed this 
project from the beginning, and we wanted each team to consider the EFF frame- 
work as a possible structure for documenting outcomes. KCALP was a partner in the 
development of EFF and learned about it both by attending EFF-sponsored training 
institutes and by integrating EFF into their program. Some Virginia programs were 
involved in EFF development, and the Mount Rogers team had been introduced to 
EFF through their state staff development system. However, they had not imple- 
mented EFF in their program. 

The Knott County team was not familiar with EFF. To introduce EFF to the 
Knott County teachers, CLS facilitators used a process the EFF staff had developed. 
Team members were asked to identify something they would like to learn to do and 
to connect these to the EFF role maps and common activities. They then identified a 
skill supporting that activity and brainstormed about ways they might practice that 
skill. For example, one Kentucky team member identified a wish to speak more 
comfortably in public. She located “Form and Express Opinions and Ideas” on the 
Citizen Role Map and “Develop and Express Sense of Self’ from the Common 
Activities, which could be supported by the generative skill “Speak So Others Can 
Understand.” She talked with another team member about ways she could practice 
this skill. 

After the Knott County team reviewed the EFF framework, each teacher 
planned to use EFF in an activity with students that would also help document 
outcomes. The lead teacher reviewed the three role maps with a class of four 
women. The group chose to focus on the citizen role. They looked at the key 
activities and brainstormed about ways to do each. They then chose one activity 
from their list: Write a letter to “Form and Express Opinions and Ideas.” This was 
an activity that they could do in one class period and that also addressed a GED skill. 
The teacher reviewed the format for a formal letter with them. Two wrote to their 
Congressman about welfare reform, and two wrote to the state’s Department of 
Transportation about bad roads. The letters served as documentation (of an output, 
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not an outcome). Other Knott County teachers tried different activities, some more 
closely tied to the EFF framework than others. Team members were pleased with 
the activities as ways to involve students in thinking about their goals and learning, 
but the activities did not result in outcomes documentation. A similar process was 
used to review EFF with the Mount Rogers team. 

Reviewing Inputs-to-Impacts 

In Stage 2, the Kentucky and Virginia teams reviewed the Inputs-to-Impacts model 
developed with KCALP in Stage 1 . Both teams added items to the original chart. 
The version on page 14 includes their revisions. 

The model had been developed as part of a process to define terms and 
clarify the relationship of outcomes to program processes. It proved useful as a 
planning and analysis tool as well. For example, as the Virginia team developed its 
documentation processes, the facilitators used the model to distinguish outcomes 
(e.g., help children with homework) from inputs (e.g., attend class regularly) in an 
early list of “learner achievements” they planned to track. KCALP used the chart to 
present its work to the other teams at the regional meeting and added categories 
describing evidence of its outcomes. (See Appendix 2). The Kentucky team’s use 
of the chart as part of the planning process is described in Chapter 4. 

Using the Inputs-to-Impacts model as a tool to review the various 
instructional activities the teams had tried led to clarification in thinking about 
outcomes documentation. The teams looked at a list of the outcomes identified from 
these activities and decided some were outcomes (e.g., helping children with their 
homework), but others were outputs (e.g., learning computer skills or doing written 
summaries). The facilitator noted at least three approaches to documentation of 
outcomes. One was to take a broad look and document whatever outcomes were 
found. The story circles and stem sentence activities did this, asking students to talk 
or write about the changes they had seen in their lives. A second method was for the 
program or teacher to identify a desired goal and document achievement. This was 
done with the calendar activity in which the teacher’s goal was for students to keep 
appointments, and she had students use a calendar to document this. A third 
approach was for the student to identify a goal and to work with the teacher to 
identify ways to determine and document when this outcome was reached. The 
teams eventually developed this last approach. 
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Two Regional Meetings 

In addition to meetings held at each program, the teams came together in two 
regional meetings. The first, held in April 1999, gave people from each team the 
opportunity to meet the other teams. Each program did a brief presentation on their 
program and community, and the team from Knox County presented their Phase 1 
work to develop outcomes documentation (see Appendix 2). Artifact 6 is one 
facilitator’s summary of the themes of the first meeting. 



Artifact 6: Reflections from a Facilitator after the April 1999 Meeting 



Briefly, the broad topics that underpinned the meeting were: 

Understanding the current situation 

Looking at what is documented, how, why 

Identifying outcomes that are not documented but are important 

Understanding state/federal systems 

Understanding the theory 

Introduction to Equipped for the Future framework 
Examining the Input-Impacts continuum 
How do we collect evidence 
Thinking about performance accountability 

More experimentation with documentation processes, focused on those particular outcomes or goals. 

Try promising processes program wide. 

We are in the process of “acting” on the process of documentation while keeping all of the above in mind. 



Discussion topics at the first meeting also included differences in reporting 
requirements among the three states and differences between what the state found 
important to document and report and what concerned students. The group also 
spent some time considering how EFF could help frame instruction that led from 
students’ goals to outcomes the students and others hoped for in their lives and 
communities. At the end of the meeting, the teams expressed excitement about 
meeting each other and about focusing on students’ needs and goals. At the same 
time, they expressed frustration at the program funders’ demands, which some saw 
as interfering with their work and requiring time to complete reports on results (e.g., 
attaining a GED) unrealistic for many students. They felt pushed to focus on things 
that were not necessarily important to the students. 

In November 1 999, the groups met again and presented the documentation 
processes they had developed and were now testing. Representatives from the 
Tennessee and Kentucky offices of adult education who were involved in developing 
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reporting systems for their states also attended and commented on the presentations. 
The state staff found the work exciting but were not optimistic that it could be 
integrated into the state performance accountability reporting. At this meeting, the 
team members also wrote evaluative comments on their project experience 
(summarized in Appendix 4). 



Summary 

In the initial meetings of both stages of this project, the action research teams took 
part in a variety of activities that increased their understanding of outcomes 
documentation. They agreed on terminology and identified the documentation 
processes already in place in their programs. They tried a variety of approaches to 
documenting outcomes and learned about EFF and how it could be used to frame 
their work. In the next cycle, the teams designed and tested their own approaches, as 
described in the next chapter. 
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CHAPTER 4: THE TEAMS’ DOCUMENTATION PROCESSES 

After the April 1999 meeting, the three teams began to focus on their individual 
documentation projects. The plan was for each team to develop a documentation 
process; pilot it with a few classes; and, if successful, move it into the entire 
program. The facilitators originally thought the programs might develop processes 
enabling them to document outcomes in ways acceptable to state systems. The 
Virginia team focused on this. The Tennessee team’s process was designed to meet 
needs identified in their program improvement process, and the Kentucky team built 
their process around particular goals the teachers and students identified. We begin 
with the Kentucky team’s results. 

Supporting Children’s Education 

The team at the Knott County Adult Learning Center was intrigued by the EFF 
framework and saw it as a way to integrate some of their students’ life issues with 
the program’s focus on academic skills. They discussed the EFF role maps with 
students, and eventually the group (staff and students) determined that they all shared 
the parent/family member role. The students listed areas of concern and parenting 
issues of interest to them. At an action research team meeting, the team sorted 
through this list and identified the overall goal of being a better parent and a subgoal 
similar to the “Supports and Encourages Child’s Education” EFF Key Activity from 
the Parent/Family Member Role Map. We used the Inputs-to-Impacts model to 
analyze this Key Activity, and the program team decided to focus on reading to 
children as a way to support children's education. Artifact 7 is the team’s planning 
grid the team created. 

Artifact 7. Planning Done on Whiteboard 

(The plain text was created by the Knott County staff and students, and the items in 
italics were added during the action research team meeting.) 



Big Goal: Be a Better Parent: Supports and Encourages Children ’s Education 



Inputs 


Individual Goals 




Processes/Outputs 


Outcomes 


Impacts 


• Children 


• Maintain 


• 


Pretest with TABE 


• Demonstrate 


• Parent/child 


• Parents 


reading skills 


• 


Develop book list 


improved reading 


transfer 


• Library 


• Do better in 


• 


Fill in calendar 


test scores 


• Better reading in 


• Teachers 


school 


• 


Folder for parent 


• Continue to use 


classroom 


• Book- 


• Parent/child 


• 


Check calendars 


library 


• Increased library 


mobile 


bonding 




weekly 


• Closer to children, 


use 






• 


Visit library 


better relationship 








• 


Talk about reading 


• Continue to read 










to children 


to child 
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The team considered teacher observation or a checklist of age-appropriate 
books as documentation but decided to have parents document their reading on a 
weekly calendar. Over the next few months, a group of parents read to their children 
and recorded what they read and the amount of time spent reading on calendar pages 
the teachers provided (see Appendix 5). Some students participated in the county 
library’s summer reading program with their children. The team used a computer 
database to record the information from the calendars, including the title of the book 
read, the source of the book (home or library), and the number of minutes read 
(5-15, 15-30, 30+). They also recorded the child’s name, age, and school, and the 
parent’s pretest TABE (Test of Adult Basic Education) score administered in the 
spring before this reading project began. 

At the end of the summer, the CLS facilitators reviewed the data and 
determined that the parents read to their children on an average (mean) of 8.8 days. 
These days were all in June because a school vacation interrupted the program. (The 
Knott County program had to move to a new center in July, causing additional 
disruption.) The team administered the TABE again in the fall. The team reported 
that 6 of the 10 participating adult students had advanced to another reading level 
on this standardized test. At the fall regional meeting, the Kentucky team reported 
that adult learners’ self-confidence had increased, their family relationships had 
improved, and their desire for their children to be readers had increased. Artifact 8 
is the summary. 



Artifact 8. From the Overhead Used by the Knott County Team at the 
November 1999 Regional Meeting 



Inputs 

• Negative past school 
experiences 

• Low self-confidence 

• Shyness 

• Lack of commitment to 
education 

• Lack of reading 
material in the home 

• Limited finances, 
inadequate childcare 
and transportation 



Processes 

Conduct orientation 
Administer pre -/post-test 
Visit library 
Develop book list 
Create reading calendar 
Provide folders 
Collect calendars weekly 
Compile data 



Outputs 

Library cards obtained 
Calendars completed 
Pre-/post-test scores 
obtained 

Number who advanced a 
reading level/showed 
improvement 
Number of books read 
Number of minutes spent 
reading 



Outcomes 

Improved reading ability 
(parent and child) 
Continued use of library 
Continued reading to 
children 

Closer relationship with 
children 

Greater appreciation of 
education 

Increased self-confidence 
Increased leadership role 
in children’s education 
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After the summer reading program ended, the team met with the parents and 
found they remained enthusiastic about focusing on their children’s education. They 
wanted to encourage their children’s reading and had ideas about how to do so. The 
team designed a new form on which parents were asked to record instances of 
reading to their children, helping with homework, children’s school attendance, 
children’s use of the public library, and meetings with their children’s teachers (see 
Appendix 5). The team planned to collect these forms monthly and record the data 
in a computer database. 

This effort was not as successful as the summer reading program. Only a 
few parents completed and returned the forms. The team decided the form was too 
complicated and not relevant to everyone. They revised it, scheduled more parent 
meetings, and loaned parents cameras to use as another way to document educational 
activities at home. The team presented the pictures at the November regional 
meeting. This was successful as a one-time effort but not something the team 
continued. 

In their final meeting with the CLS facilitators, the Knott County team said 
the documentation work had helped them know their students better. They also said 
both students and staff were more deliberate and purposeful in their work. They 
described their changing student population and how this affected their docu- 
mentation efforts. When the action research project began, their students were 
mostly mothers in a welfare-to-work class. Several of the students in the parent 
group had graduated or left the program. The program’s students now included 
many more young male and female students who were not parents. The focus on 
supporting children’s education was no longer as appropriate. After the Docu- 
menting Outcomes project was “officially” over, the Knott County team continued 
to experiment with EFF and with documenting their work. They have focused on 
developing a process to help students set and document goals. 

Taking Responsibility for Learning 

In Tennessee, KCALP team teachers had developed processes to document outcomes 
during Stage 1 but had not carried them beyond their individual classes (see pp. 
15-16). After the April regional meeting, the Virginia and Kentucky teams focused 
on developing their documentation processes, but KCALP went through a crisis 
period brought on by conflicts between county and state agencies involved in 
providing educational services to welfare-to-work clients. The program survived 
this crisis, but KCALP no longer contracted to provide these services, and several 
teachers left. By the end of the summer, the situation had stabilized, and KCALP 
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was ready to plan for a new documentation process. Three of the original team 
members and four new teachers participated in the action research team in Stage 2. 

Throughout the fall, the team met and developed a plan to document the 
outcomes addressing the EFF Standard “Take Responsibility for Learning.” They 
presented the plan at the November regional meeting (Appendix 6). Plans included 
teaching about the standard, using a story that demonstrated someone taking 
responsibility for learning, and collecting data on students taking responsibility for 
learning. 



Figure 1. EFF Standard “Take Responsibility for Learning” (from Stein, 2000) 



STANDARD 



Take Responsibility 
for Learning 

• Establish learning goals that are 
based on an understanding of one's 
own current and future learning 
needs. 

• Identify own strengths and 
weaknesses as a learner and seek 
out opportunities for learning that 
help build self-concept as a learner. 

• Become familiar with a range of 
learning strategies to acquire or 
retain knowledge. 

• Identify and use strategies 
appropriate to goals, task, context, 
and the resources available for 
learning. 

• Monitor progress toward goals 
and modify strategies or other 

; ' : features of the learning situation 

I ^necessary to achieve goals. r 
n«wl iMrhln^^ to ^ : 






- + ; 
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Teachers gave students a presurvey on Take Responsibility for Learning 
(TRL), asking them to write briefly: 

• What does it mean? 

• Why is it important? 

• How can I use it? 

A postsurvey asked similar questions: 

• What does it mean to take responsibility for learning? 

• How does taking responsibility for learning work? 

• Is taking responsibility for learning important to you? Why? 

• How have you used responsibility for learning in your life? 

• How can you use responsibility for learning in your life in the future? 

During the month between the surveys, teachers focused on TRL, introducing 
the concept by asking students to read, write about, and discuss a story about a 
famous person (e.g., Bill Cosby) who took responsibility for learning in his or her 
life. Students were asked to keep a journal of events in their daily lives that indi- 
cated taking responsibility for learning and to share these with the class. At first, 
students tended to write the same things every day and did not seem to grasp the 
TRL concept. The teachers tried brainstorming about TRL’s meaning in a staff team 
meeting and then did a similar activity with students. These brainstorming activities 
seemed to improve students’ understanding of and commitment to the project. The 
teachers also kept logs of their observation of TRL in the classroom, though they 
reported they did not really have the time to record everything they noticed. 

The KCALP team met with CLS facilitators and discussed their experiences 
with using the TRL standard. They found the student journals were useful as a 
writing activity, and the teacher log helped with planning. However, the pre- and 
postsurveys were most useful in identifying and documenting instances of students 
taking responsibility for learning both in class and in their everyday lives. Among 
the changes they noted in students were: 
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• Increased writing in journals and greater variety in examples of TRL that 
students identified 

• More learning activities outside class 

• Some students were more aware how they best learned 

• Two students got e-mail accounts, and two got library cards 

• Students became better at setting short-term goals 

• Some students were more proactive about issues in their lives (e.g., getting a 
landlord to make repairs or asking for a work schedule that fit with their 
schooling) 

The teachers also reported changes in themselves. As they worked on the 
project, they adapted their own classes (e.g., using facilitated discussion in a 
beginning literacy class to identify instances of students taking responsibility for 
learning). One teacher said she became “less authoritarian” and “included more 
independent and flexible student-centered learning time.” The project reinforced the 
team’s commitment to involving students in decision-making. The project also 
served as a team-building process for a group of new teachers. KCALP plans to 
continue to teach TRL, introducing this EFF standard as part of its month-long 
Learning Skills orientation class. As the program implements EFF, TRL will be 
applied in student goal-setting, planning, and assessment processes. 

DoSetMet 

From the beginning of the project, the Mount Rogers team focused on connecting 
action research work with efforts to develop a process for reporting required data to 
the state. They reviewed Virginia state documentation requirements as well as the 
forms used locally. They then developed a new form to be used at student intake to 
collect the information required by the state. This included demographic information 
and the student’s source of information about the program, reasons for enrolling, 
goals, test scores, and other information. They also added a list of “learner 
achievements” based on a short checklist of personal, social, and academic learning 
skills. These were chosen as a way to begin to document more than test scores. 

Some were outcomes (e.g., “helped child with homework”) and some were class- 
room activities (e.g., “worked on assigned tasks”). After discussion in action 
research team meetings and several revisions, the team decided to move their 
outcomes list to a separate document. 
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The team had determined that using activity-based documentation, such as 
story circles and stem sentences, was too time intensive. Instead, they developed 
their outcomes document as a checklist, using some items from their learner 
achievement list. They first conceived of the checklist as an exit instrument and a 
supplement to the other reporting form. It was organized around the three EFF roles, 
plus the category of “self.” The original draft had a space to check when something 
was accomplished and a space for comments. The following were examples of 
accomplishments: use library, ask for directions, read help-wanted ads, use a 
computer, volunteer in child’s school, and pay bills. The team decided to add a goal- 
setting column to use at intake. The learner could note which items she or he had 
already accomplished and those that would indicate progress toward goals. Items 
relevant to the student’s goals would be the focus, but other items accomplished 
could be checked as well. Though these were decontextualized competencies or 
activities, they could be contextualized by the learner’s goals. The team discussed 
having a space for the goals on the form and giving the student a copy of the form, 
both to facilitate documentation and as motivation. 

The team tried using this new form with several students. They noted student 
reactions, how long the students’ took to complete it, and whether students could 
give evidence of their accomplishments. They found that although it took some time 
to complete, most students liked the form and were able to describe their accom- 
plishments. The students added items and suggested language changes. The team 
found the list helped both the teacher and the students think about goals and out- 
comes. The form was revised to include student suggestions. In its current iteration, 
the form lists 43 items with three possible responses to each item: currently do (DO), 
would like to do (SET), and now can do (MET). Artifact 9 reproduces the summary 
version of this form. 

In fall 1 999, several Mount Rogers teachers outside the team tested the 
DoSetMet form. These teachers met with the team in November to reflect on their 
experience with the form. In this group meeting and in written evaluations, the 
teachers were quite positive about the form’s usefulness both as a way to help them 
know their students and as a goal-setting process for students. Several noted the 
form helped build students’ self-esteem as they focused on what they could do. An 
employee of the Virginia Department of Human Services who attended the meeting 
requested a copy of the form to use as part of the department’s intake procedure for 
new clients. 

Virginia had introduced a statewide computerized data collection system 
before it was required for the National Reporting System. One of the data items was 
achievement of learner goals. Mount Rogers teachers used the DoSetMet form to 
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document learner goals as students set and achieved them. This information could 
then be entered into the information system. Virginia recently revised its reporting 
system, and identification and documentation of student goals is no longer a focus. 
However, Mount Rogers teachers continue to have DoSetMet as an option to use for 
goal setting and documentation; some teachers, including the action research team 
members, continue to use it. But the Mount Rogers team has not been able to 
integrate the form into their formal reporting system as they had hoped, and form 
has not been introduced statewide. 
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Artifact 9: Do/Set/Met 

DO/SET/MET GOAL SHEET 

Instructor: Class: 

Date: 



Please fill in the total number of students in your class that have checked DO/SET/MET in each of the following 
categories. 



/- Worker 


Do 


Set 


Met 


Comments 


1 . Fill out a job application 










2. Arrive at appointments on time 










3. Fill out work-related forms 










4. Read and understand work related materials 










5. Communicate with others in work/social settings 










6. Have a job 










7. Obtain job promotion 










8. Organize, plan, and prioritize work 










9. Use a computer 










II - Citizen/Comm unity Worker 










10. Use library 










1 1 . Locate and/or use community agencies or services 










12. Have a driver’s license 










13. Use public transportation 










14. Complete U.S. citizenship class 










15. Apply for legal immigrant status, U.S. citizenship, emancipation papers 










16. Register to vote 










17. Vote in primaries/elections 










1 8. Obtain legal advice 










19. Use maps 










20. Ask for directions 










21 . Active in volunteer work 










22. Participate in neighborhood watch/activities 










III - Parent/Family Member 










23. Read to children 










24. Help children with homework 










25. Volunteer in child’s school 










26. Talk with teachers/school staff 










27. Attend school-related meetings (e.g., PTA) 










28. Fill out insurance forms 










29. Use a recipe 










30. Assemble a toy/equipment/fumiture 










3 1 . Read product/medicine label, directions, and safety warnings 










IV -Self 










32. Read daily (magazines, books, newspaper) 










33. Have a checking/savings account 










34. Balance a check book 










35. Have a personal/family budget 










36. Pay bills 










37. Use an ATM card 










38. Count money 










39. Compare prices to determine the best buys 










40. Solve a problem 










41. Feel good/better about myself 










42. Feel independent 










43. Meet a personal goal 











This form was developed by Rita Roper, Jerry Musick, and Sherri Whitlock, teachers in the Mount Rogers Regional Adult 
Education Program in Abingdon, Virginia. 
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Summary of Team Activities in Stage 2 

Three diverse programs participated in the Documenting Outcomes action research 
project. They all developed outcome documentation processes they found useful, at 
least in the short term. Table 2 summarizes their efforts. 



Table 2. Summary of Stage 2 Team Activities Developing Outcome 
Documentation 





KCALP, Tennessee 


Mount Rogers, Virginia 


Knott County, Kentucky 


Documentation 

Process 


• Five teachers spent a 

month working with their 
students focusing on the 
EFF standard Take 
Responsibility for 
Learning (TRL) in 
students’ lives. 


• Three teachers developed 
a checklist of 43 possible 
goals/outcomes that . 
students use to evaluate 
current skills, set goals, 
and document 
accomplishment. 


• Five staff members 

worked with students to 
identify the goal to 
support their children’s 
education. Students 
documented activities 
that did that, e.g., reading 
to their children. 


Documentation 

Purpose 


• To determine how 
students were 
implementing TRL in 
their daily lives 


• To help students set and 
document goals for state 
reports. 


• To motivate students and 
document increased 
activities supporting 
children’s education 


Documentation 

Tools 


• Pre- and postsurveys 

• Student Journals 

• Teacher observations 
recorded in journals 


♦ DoSetMet checklist 


• Calendar forms 


Where 

Implemented 


• In program 


• Countywide 


• In program 


Used 


• As a foundation skill for 
learning 


• To report goal 
achievement to state 

• To plan instruction 


* Asa focus for 

instructional activities 


Continues 


• In Learning Skills 
orientation class 


• As an option, countywide 

• Team teachers are using 


• No 
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CHAPTER 5: FINDINGS: WHAT DID WE LEARN? 

Action research is a spiral of research, learning, action, and reflection. It is an 
ongoing process without a clear point at which the researchers come to the end of the 
project, analyze their data, and report their findings. As practitioners research their 
own practice, analysis is ongoing, questions change, and the project doesn’t really 
end. But at some point, the focused work stops, and researchers can summarize what 
has been learned. We have reached that point with this report. In this section, we 
begin to answer the following questions: 

1 . What have we learned about documenting outcomes of participation in adult 
basic education programs? 

2. What have we learned about conducting action research in adult basic education 
programs? 

3. What have we learned about how action research affected the participants? 

The CLS team developed the findings in this section on the basis of an 
extensive review of project data, including team and regional meeting minutes, 
artifacts from program teams, CLS staff field notes and personal reflections on 
meetings, results from evaluations, interim and final team reports, and informal 
interviews with team members. 

Documenting Outcomes of Participation in Adult Basic Education Programs 

In this project, we set out to work with adult basic education programs to develop 
indicators and measures that could be used to document the outcomes of student 
participation in adult education. We were interested in going beyond what could be 
measured by intake/exit demographic data and standardized tests. We hoped 
program teams would be able to document what one teacher called the “invisible 
outcomes,” the changes teachers saw or heard about from their students but had no 
way to report. We hoped that, with the project participants, we could create methods 
to capture and report these outcomes as part of state performance accountability 
systems. We imagined being able to document outcomes in a way that made a 
compelling case for community impact. 

The teams that took part in this action research developed ways to document 
changes in students. Some of these changes fit our definition of outcomes and go 
beyond classroom activities — they are changes that make a difference in students’ 
lives. The Virginia team’s DoSetMet form established a process that helped students 
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identify the changes they hoped to make (or their educational goals) and document 
achievement of those goals, or outcomes. The Tennessee team documented 
outcomes as well as outputs as part of their focus on EFF’s TRL Standard. The 
Knott County Adult Learning Center students used calendars to document activities 
that supported their children’s education, such as reading to their children and 
meeting with teachers. 

These documentation efforts focused on particular parts of students’ lives that 
the program or students identified as areas in which they hoped to make change. The 
three teams developed ways to document changes the students reported and the 
teachers observed. The teams, their programs, and their students have found these 
documentation efforts useful tools for instructional planning and for learner and 
project assessment. The programs have continued to use these documentation tools 
and to develop others. 

For the participating teams, this action research project has led to increased 
understanding of how programs might identify and document the outcomes of adult 
education participation in learners’ lives in ways that meet local program needs. 

This project explored, through action research, the development of indicators and 
measures of the impact of participation in adult literacy programs and used the EFF 
framework to inform this work. 

However, in the span of this project, we were not able to develop measures 
acceptable to the state accountability systems with which we hoped to connect. 

This was not caused by the involved states’ lack of interest. State adult basic 
education staff from all three states followed their state team’s work and engaged in 
at least some consideration of how it might be integrated into state adult education 
reporting systems. Unfortunately, the federal performance accountability require- 
ments limited use of locally identified and documented outcomes. The state staff 
were more concerned with establishing the reliability and validity of locally devel- 
oped documentation. Putting in place a new reporting system that would meet the 
guidelines of the National Reporting System was their priority, and they did not feel 
they had the resources or perhaps the latitude to experiment with other approaches. 

For local documentation efforts to be part of a state performance account- 
ability system, local programs need a commitment to rigorous, ongoing data 
collection and analysis. Each process the action research teams developed included 
collection of some baseline data (on goals, test scores, or assays of knowledge of a 
particular EFF standard) and collection.of data on outcomes. However, analysis of 
this data varied from project to project, most likely because of the facilitators’ 
limited focus on analysis and the teams’ limited time. Giving the teams more 
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extensive training in analysis as well as a longer commitment of time would have 
been helpful. 



The Rewards and Challenges of Action Research in 
Adult Basic Education Programs 

Participating in this action research was both a positive experience and a challenge 
for the people involved. In this section, we examine what was most helpful and 
valued by the participating teams, the challenges the teams and facilitators 
encountered, and particular issues we faced as facilitators. 

The Rewards 



Throughout the project, we asked for the teams’ feedback on the action research 
process through informal activities, such as having them write brief answers to 
questions on cards, and through structured surveys completed at regional meetings 
(see Appendix 3). In these activities and surveys, the team members identified a 
variety of activities they found useful, including analyzing current documentation in 
the Documentation Matrix and using the Inputs-to-Impacts chart. More of the 
positive comments concerned the overall process. People liked working in groups 
and having a chance to talk about their work and share experiences with teachers on 
their own and other teams. As one teacher wrote: 

I enjoy listening to the ideas/opinions/responses of my coworkers. It helps me 

understand their focus. 

Several people also mentioned appreciating having facilitators who brought 
different ideas and different approaches to facilitation. One person said she found 
“respect for everyone involved.” 

The two aspects of this project the participants seemed to particularly value 
were the focus on students and the opportunity to reflect on the goals of their work. 
The focus on students and their goals influenced how team members approached 
their work, as discussed in Chapter 5. 

The Challenges 

As the facilitation team reflected on what we learned from this project, one of us 
summed up the challenges by saying, “This work is hard.” And it was, because of 
the process and for reasons that are part of working in adult basic education. 
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We used an action research methodology because we wanted to work with 
programs to develop outcomes documentation rather than field test a particular 
approach. But this led to some initial frustration for team members, who noted, 
“There were no predefined test instruments in place” and “We didn’t know what to 
collect or what to do with it.” In the first stage, we worked with the Knox County 
Adult Literacy Program to conceptualize what we were doing: What did we mean by 
“outcomes”? Who was the documentation for? How did our documentation 
processes fit with the program and state processes? In this first stage, we were able 
to develop processes that helped the Stage 2 teams go through similar thinking in a 
more structured way. But these teams also felt some initial frustration. Helping 
everyone involved in the project understand and, if possible, appreciate the processes 
of developing the action steps to be taken is one of the challenges of action research. 

When the teams were asked to identify challenges, they named some 
conceptual issues, such as “the idea of documenting changes in learner lives versus 
test scores,” but most named practical challenges, including: 

• Getting students to provide data by completing journals, calendars, and charts 

• Student turnover and a changing student population with more young students 

• Limited teacher time to design, collect, and use documentation 

• Difficulty in finding ways to share results with other teachers in other programs 

• Specific program issues, such as office moves or funding changes 

Many of the project’s challenges are those of our underfunded field. In 
discussing the amount of personal (uncompensated) time her teachers devoted to the 
project, one administrator commented, “This is not a new problem for adult edu- 
cation — it is a continuing problem that is a barrier to providing excellent education.” 
Most of the teachers were part-time. The marginal place of adult education in the 
larger system is typified by the forced relocation of the Knott County program and 
the loss of major funding in Knox County. 

The changing student population the action research project teams noted is 
typical. In adult basic education, there is high student turnover (Young, Fleischman, 
Fitzgerald, & Morgan, 1995) and an increasing number of younger students (Hayes, 
2000). Documentation of outcomes assumes continuing contact with students or at 
least an exit interview, but this is difficult to obtain, as students often stop attending 
class without notice (Beder, 1999). 
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The requirements of the performance accountability structure developed in 
response to Title II of the Workforce Investment Act (1998) also had implications for 
local program teams. State reporting requirements were in flux. The pressure and 
confusion of adopting a new reporting system added to the teams’ work. The 
mandated performance accountability system seemed to inhibit states’ willingness 
to experiment with the processes that the action research teams were developing. 

The Facilitation Process 



As facilitators, we increased our knowledge of action research as a learning process. 
We came to realize that coming to consensus was part of any action research. Our 
growing understanding about the connections between documentation and other 
program processes, as well as our understanding of the challenges programs and 
teachers face in trying to integrate documentation with instruction, came with 
reflection on the ongoing work. 

One issue in action research in adult education is team members’ 
participation in all aspects of the research process (See Adelman, 1997; Peters, 

1997). We did not begin this project thinking of it as participatory action research. 
As facilitators, we had already defined the question we wanted to address, and we 
established the structure of the process. But as the project progressed, we found it 
was in many ways controlled by the teams. We had been clear from the beginning 
that the documentation processes the programs would implement would meet their 
programs’ needs. Although we, as the researchers, could have identified particular 
measures of particular outcomes and asked the programs to pilot these, our approach 
was to work with the program teams to develop measures— or processes — that fit 
their contexts. The work sometimes proceeded in unexpected ways and was tied 
more than we might have anticipated to particular instructional goals, but this was 
what was needed and possible. The teams’ work also led to results beyond the 
original intention of the project, as discussed in the next section. One facilitator 
described our experience: 

The hardest part and the thing that sometimes makes the process awkward is not 
having a “playbook” because the process is organic, but the unknown, the twists and 
turns of this kind of work, is where the yield that is the most informative and 
ultimately the most gratifying comes into focus. 

Another quandary we faced was how much we could reasonably request of 
team members. Although they received compensation for time spent on the project, 
they had too little time. In retrospect, we might have, for example, asked for more 
written documentation of team meetings we did not attend. We might have 
established a more extensive documentation process for the project, but we were 
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hesitant to push for too much. The teams worked hard and enthusiastically, and their 
investment in the project may compensate for what we lost in data. 

Looking back on the project, we have identified several factors we might 
change in another cycle. First, as facilitators, we needed more time on the project. 
Like the program teams, we were all part-time. The teams were in three different 
states, which limited our ability to attend all team meetings. We also felt limited by 
funding and did not extend the programs’ involvement more than a few months 
beyond the original plan. In retrospect, it would have been useful to have had more 
time to work with the program teams to help them refine and analyze data from their 
documentation processes. 

Although the facilitation process was not always smooth, we developed tools, 
such as the Documentation Matrix, that we found quite useful in helping the teams 
look more broadly at their programs and their practice. We came away from this 
project with a renewed commitment to the value of action research in sharpening the 
questions participants ask about their work and the work of adult basic education as a 
system, and in giving participants tools to help answer their questions. 

What Else Have We Learned? Action Research as a Learning Process 

In his article “The Role of Research in the Practice on Adult Education,” Allan 
Quigley (1997) places action research in the “practical” category of his “Research 
Intentionality Framework” (p. 1 7), or research for “practitioner development and 
institutional improvement.” Although the purpose of our research was to contribute 
to understanding a systemic issue — how to document the outcomes of participation 
in adult basic education — we found it also contributed to participants’ professional, 
program, and personal development. 

Professional development 

“The result is improvement in what happens in the classroom and school, and a 
better articulation and justification of the educational rationale for what goes on” 
(Kemmis & McTaggart, 1984, p. 5, quoted in Quigley, 1997). The teams reported 
these kinds of results in their final reports: 

• The project built a team out of a group of newly hired teachers 

• The focus on TRL led the teachers to give students more responsibility 

• Teachers and students were better able to set goals 
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• Teachers focused more on student goals 

• Teachers know students better and are better observers of students’ life situations 

• Teachers and students use different thinking processes and are more deliberate 
and purposeful 

• Teachers are able to turn new ideas/approaches into teaching strategies 

Effective professional development involves engagement in active thinking 
about one’s practice, and this occurred during this project. As they participated in 
this action research, team members changed their understanding about aspects of 
their practice and reported ways their practice had changed. The process of 
identifying desired outcomes led to increased instructional focus on meeting learner 
goals and achieving desired outcomes. As one teacher said, “If you start with 
student goals and you document reaching those goals, you better do something in the 
classroom to help them learn what they need to get there.” In some instances, this 
meant a classwide focus, as in the Knott County summer reading project; in others, it 
meant extra focused help from a tutor as a student worked to meet a goal. 

Developing processes to document changes in learners’ lives meant talking 
to learners about topics at a depth new to many team members. They found they 
gained understanding of their students’ lives and the barriers students faced. One 
teacher who piloted the DoSetMet form spoke of how it led her to initiate dialogue 
with students, and the team members from Virginia said that having a tool — the 
DoSetMet form — facilitated their ability to talk to students about issues in the 
students’ lives. A Tennessee teacher reported, “Based on the issues and concerns 
revealed by this project, I revamped my classroom procedures to include more 
independent and flexible student-centered learning time.” Ten of 13 team 
respondents to a survey late in the project said their instructional planning and 
practice now focused more on student goals and needs. 

As part of the project, team members also had opportunities to try new 
instructional techniques and learn research skills of data collection and, in one site, 
computer data entry. One Kentucky team member commented that she had 
“stretched” herself. One final team report spoke of the staff being “able to turn 
new ideas/approaches into teaching strategies.” 
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Student development 

This project also led to learning opportunities for students. All three teams involved 
students in designing documentation processes. The Virginia team tested each 
version of what became the DoSetMet form with students and revised both the 
content and the language on the basis of student suggestions. The Kentucky team 
focused their project on parents’ reading to children after the team used the EFF 
framework in a series of meetings with students to identify a goal (to support their 
children’s education) and steps to meet that goal. The Tennessee team revised their 
approach to documenting TRL on the basis of student feedback. 

Students reflected on goals and accomplishments in each of the teams’ 
projects. The Mount Rogers team reported that the process motivated students and 
built their confidence. Students also set more specific goals when they used the 
DoSetMet form. The Knox County team also reported that students were better at 
setting short-term goals. 

In addition to the goal-setting changes and an increased awareness of 
accomplishments, project activities generated a variety of student outcomes and 
outputs. The Knott County team reported improved reading, increased library use, 
and better “organizational habits.” Some Knox County students developed an 
interest in writing in a journal and continued to do so after the TRL project ended. 
The team reported, “Almost all students seemed to have learned more about 
themselves as learners.” The learning activities that led to these changes could, of 
course, be carried out as part of regular classroom practice, but the action research 
project encouraged the teams to undertake these activities. 

Program development 

The Knox County team has been intentional about program improvement for several 
years. They are participating in the Baldridge National Quality Program award 
process and have set clear priorities for change. Participation in the action research 
contributed to this process, helping them “understand what they were doing and 
why.” One KCALP team member felt affirmed in the program improvement effort 
after observing the action research facilitators’ similar processes of reflection and 
analysis. For all three teams, thinking through program processes led to increased 
appreciation of how different program aspects — goal setting, instruction, outcomes 
documentation — can be aligned. 
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Personal development 

All team members reported that participation in the action research project changed 
their classroom practice. For some, the changes went beyond the classroom. One 
administrator told us the project had helped teachers think of themselves as actors 
outside their own classroom. They saw they had knowledge and understanding to 
contribute to solving program issues and, by extension, issues of concern to the field 
of adult education. Several have presented at state conferences and written for 
newsletters. Although we cannot attribute their activism to the action research 
project, it seemed to support it. 

One team member attributes a major change in her life — returning to 
college — to her participation in the project. She speaks of the project as changing 
her life because it convinced her that she could further her own education. 

Summary: What Did We Learn? 

From the Documenting Outcomes action research project, we have learned: 

• Local adult basic education programs can develop documentation processes 
useful for planning and assessing their work 

• More work is needed at the local, state, and national levels before locally 
developed documentation processes can be used for performance accountability 
systems, such as those required by the Workforce Investment Act 

• Although practitioners value participation in action research, such factors as the 
limited paid time for teachers typical of adult education in most states make it 
difficult 

• Action research is effective for professional and program development in adult 
basic education 
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CHAPTER 6: IMPLICATIONS AND FINAL THOUGHTS 

Action research probably has most relevance for the programs in which it occurs. 

But the action research described in this report also has wider implications for adult 
basic education practice and policy. We have identified implications for professional 
development, program improvement, and performance accountability. 

Action research is a valuable professional development tool in adult education. 

As they participated in this action research, the team members changed their 
understanding about aspects of their practice. The work of understanding and 
developing outcomes documentation created opportunities for reflections that led to 
improved classroom practice. Developing processes to document changes in 
learners’ lives meant talking to learners about topics and at a depth new to many 
team members. They found they gained understanding of their students’ lives. The 
process of identifying desired outcomes led to an increased instructional focus on 
meeting learners’ goals and achieving the desired outcomes. Thinking through 
program processes led to increased appreciation of how different program aspects — 
goal setting, instruction, outcomes documentation — can be aligned. Team members 
also gained a greater understanding of research and greater awareness of research as 
a source of knowledge that might contribute to their work. Although some changes 
the project teams reported might be specific to a focus on outcomes documentation, 
the participants’ experiences indicate that action research in which the question and 
methodologies are in part determined by others can serve as valuable professional 
development. 

The adult education literature has only begun to discuss action research as a 
professional development tool, most extensively in Quigley and Kuhne’s 1997 New 
Directions edition: Creating practical knowledge through action research: Posing 
problems, solving problems, and improving daily practice. The adult education 
literature supports using practitioner research (also referred to as practitioner inquiry) 
as part of professional development (Drennon, 1994; Fingeret & Cockley, 1992; 
Lytle, Belzer, & Reumann, 1992). Typically in practitioner inquiry, an individual 
teacher identifies and investigates a question of concern to him or her. Teachers may 
be part of a research group, but their question and investigation is usually their own. 
Action research could extend these efforts by including group research on questions 
identified by both programs and other entities (e.g., the EFF field research in which 
the work of collecting data for EFF standards’ development and the accompanying 
assessment framework is also changing teachers’ practice; Stein & Bell, 2001). In 
K-12 educational literature, action research is recognized as an effective approach to 
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professional development (See, for example, Altricher, Posch, & Somekh, 1993; 
Zeichner, 2001). 

Recommendation 



Those responsible for professional development in adult education should use action 
research more extensively. By doing so, they can learn from the experiences 
reported here, from the EFF work, and from the states where action research has 
been used in professional development, particularly Pennsylvania and Tennessee 
(see, for example, Quigley & Weirauch, 1997; Action Research Group on Learning 
Disabilities of the Center for Literacy Studies, 1994). To effectively improve 
practice, teachers need to be paid for time spent in action research, the state system 
needs to accept action research as professional development, and facilitation support 
needs to be available. 

Action research is a tool to help student set and monitor goals. 

Students set particular goals and helped identify the particular outcomes that were 
documented in this action research. In their research on learner persistence, 
Comings, Parrella, and Soricone (2000) identified student goals and goal-setting as 
important to supporting learner persistence in adult education. Involving students in 
action research to identify goals and develop processes to document goal achieve- 
ment may have a positive impact on student persistence. 

Recommendation 



Action research involving students in both setting and documenting achievement of 
their goals should be pursued and evaluated as a tool to further learner persistence. 

Action research can enhance processes for improving program quality. 

Action research focusing on outcomes can contribute to building local program 
quality by supporting systematic thinking about what the program does and why. 
Establishing goals and examining outcomes focused the action research teams in this 
project on the processes that lead from goals to outcomes. The cycle of observing, 
planning, acting, and evaluating fits such approaches as the Baldridge National 
Quality Program (National Institute of Standards and Technology, 1998) and Project 
Equal, Pennsylvania's field-based program improvement initiative (Alamprese, 
2000 ). 
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Recommendation 



Local and state ABE administrators should encourage use of action research 
approaches to improve program quality. Systematic processes of reflection to 
identify areas that need improvement, combined with ongoing action and evaluation, 
help keep a program focused on continuous improvement. Resources such as How 
are we doing? (Bingman, 2001), a guide for local programs based on this action 
research project, can be used to facilitate local inquiry into program improvement, 
particularly when local programs have access to financial support for staff time. 

Performance accountability could be supported and enhanced by action 
research. 

The Workforce Investment Act mandates “a comprehensive performance account- 
ability system” (Section 212) that states are to use “to assess the effectiveness of 
eligible agencies in achieving continuous improvement of adult education and 
literacy activities.” In her review of performance accountability in adult basic 
education, Juliet Merrifield (1998) recommended four principles that support 
effective performance and accountability in adult literacy and basic education. 

These may be summarized as: 

• Agree on performance by coming to consensus on goals and what is to be 
measured as indicators of accomplishment of those goals 

• Develop mutual accountability relationships so communication and support 
move in all directions within the system 

• Build performance capacity and be accountable so programs have needed 
resources to meet goals and the capacity to measure achievement 

• Create new tools to measure performance so multiple measures are used and 
instruction is not targeted at a few easily measured items 

The action research conducted for this project addressed each of these 
principles. The processes involved the teams in clarification of their goals and led to 
consensus on at least some of the performances they hoped to measure. The action 
research seemed to strengthen mutual accountability at the program level. Teachers 
expressed a new understanding of students’ needs and a more focused effort to meet 
particular needs. For students, identification of goals seems to deepen commitment 
to their learning, which is also a finding of other studies (Comings, Parrella, & 
Soricone, 2000). Documenting the activities in their lives that supported their goals 
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also built commitment. Research on ways to document outcomes built the capacity 
of the teams to be accountable — that is, to document performance on at least some 
goals — and new tools were created. 

But local efforts such as those described in this report cannot by themselves 
build a comprehensive performance accountability system. The work must extend to 
the state and national level if the principles Merrifield outlines are to be achieved. 
For example, a more extensive and focused process of examination and consensus 
building around goals involving all the programs in a region or state could undergird 
a state performance accountability system. Mutual accountability should go beyond 
teachers and students to include state and federal agencies. Systems of feedback and 
accountability need to be put in place. For programs to have the capacity to truly be 
accountable for both measuring and meeting goal achievement, the challenges 
identified in this project, particularly limited staff time, will have to be addressed. 

More extensive projects that involve more teams in a program or state have 
the potential to build a system of accountability that integrates a variety of tools to 
measure the performances that are recognized as most important. Local programs 
can develop new ways to measure performance, but integrating these processes into 
state reporting systems will require changes. The federal National Reporting System 
and most state systems require standardized measures of only a few outcomes. 
Increased flexibility on the part of state and federal policymakers is needed so that 
locally developed processes for documenting a wider variety of outcomes can count 
as measures for program accountability. And more complex and nuanced systems 
must be developed and used to report the kinds of data collected by teams that take 
part in projects such as this. Although national legislation focuses on economic 
outcomes of adult education, learners have a wider variety of goals. Programs need 
to have the ability to focus on these individual goals as well as nationally established 
goals. As the EFF Assessment Framework takes shape, with a clearly identified 
performance continuum for each of the 16 standards, local programs maybe able to 
use the EFF standards as the vehicle for identifying, assessing, and reporting student 
goals and progress. 

Recommendations 



• States should build consensus on the goals that are the basis of their performance 
accountability systems by using participatory processes such as action research. 

• Action research should include some focus on locally defined goals as part of 
consensus building because a focus on goals and measurement of goals’ 
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achievement seem to build programs’ ability to implement performance 
accountability systems. 

• State and federal resources should be committed to designing outcomes 

measurements and reporting systems flexible enough to include a variety of goals 
as well as rigorous enough to measure performance. These might include 
performance-based assessment frameworks, such as that of EFF (Stein & Bell, 
2001), and Web-based reporting systems that allow reporting of specific 
evidence of goal achievement. 
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APPENDIX 3: Sample Facilitator’s Agenda 

Documenting Outcomes for Learners and Their Communities 
Action Research Team Meeting 
Knott County Adult Learning Center 
August 27, 1999 

Facilitator’s Agenda 



Check In 

Review project status 

1 . Rest of the fall developing and implementing documentation processes of 
outcomes under broad goal of Support Child’s Education 

2. CLS development of inquiry guide 

3. November 19th meeting to share work and evaluate learning, processes, advise, 
suggestions on using for continuous improvement 

Report from August 16 meeting re: KY WIA documentation requirements — what do 
they know now? 

Report on my conversation with Sandy K. 

Review work with parents 

Report from team on meeting with parents. Is there still a commitment to focus on 
parent role? 

Have they identified additional outcomes? 

If this hasn’t happened, see if it would be helpful to plan a parent session together. 
We could assist or just help plan. It would be good for us to have this for our later 
product. 

If it has happened, what are the additional outcomes? 

Documenting additional outcomes 

Look at the outcomes (either determined with parents or some the team thinks are 
likely). Eventually need to identify 4-5 to track. 

What might be evidence of progress? Look at EFF standards for connections. How 
could this evidence be collected? The Summer Reading Project (SRP) fits into this. 
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This “evidence of progress” are outputs. The reading to children was an output of 
focusing on reading to children — and encouraged the reading. 

What are the outcomes? For example, for the SRP outcomes might include: 

• Parents developing the habit of reading to children (evidence? documentation?) 

• Parents and children regularly using library (evidence? documentation?) 

• Improved reading skills for parents (evidence? documentation?) 

Impacts of SRP and other work should include school success. 

Summer Reading program data 

Look at data display from June calendars. What do they note? Does it suggest any 
changes in documentation or instructions? 

Next steps 

Either proceed with parent meetings or develop more documentation of progress and 
outcomes processes/forms 
Set implementation of process 
Set evaluation 
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APPENDIX 4: Summary of responses from team members at the 
11/19/99 regional meeting (N=13) 



What has changed about 
your instructional 
practice and planning? 


• More focus on student goals and needs (10) 

• More real life materials, situations (4) 

• Some changes in planning (3) 

• Integrating learning activities/processes around EFFATake 
Responsibility for Learning (3) 


Do you collect more or 
different evidence/ 
items/things from 
students? 


• More questions to individual students about their lives 
(oral/written) (7) 

• Collecting evidence that students achieved specific 
goals/outcomes (4) 


What was the place/use 
of the EFF framework in 
what you did? 


• Basis/foundation/center of all our work (5) 

• EFF skills (7) and roles (4) are a part 


What were the things 
that made this difficult? 


• No assessment in place, hard to document changes (7) 

• Student attendance (2) 
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APPENDIX 5: Knott County Team Family Reading Project Forms 



Summer Reading Project 



Name: 


Child/Children’s Name 


Age 


Sex 


Attended School 


































Pre-Test Score: TABE Post-Test Score: 



DATE 


DAY 


TITLE OF BOOK 


ORIGIN 


MINUTES 


06/19/99 


Sat 


The Cat in the Hat 


Home 


5-15 
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APPENDIX 5 (continued) 

Knott Co. Adult Learning Center 

EFF Parent/Family Member Role : Goal/Key Activities - 
Foster informal education of children/Support children’s formal education 



NAME: MONTH: 





M 


T 


W 


T 


F 


M 


T 


W 


T 


F 


M 


T 


W 


T 


F 


1 . Read to 
Children 








































2. Help with 
Homework 








































3. Good 
Attendance 








































4. Library 
Use 








































5. Meet with 
Teacher 
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APPENDIX 6: Knox County Plan 

Knox County Adult Literacy Program 
Action Research Project 



PLAN: 

Question: How do programs capture changes in the lives of students? Do they 
take responsibility for learning and how does it look in their daily lives? 

Data to Collect: 

Student writings (in a journal) or tape recordings of events that occur when they are 
at home (not at school). They will answer the questions: What, When, Where, Why, 
Who, How? 

Students will use the EFF skill wheels to show skills they used. The journal and skill 
wheels will be brought into class daily and discussed with the teacher. 

Teachers will keep their own journals that reflect their observations of TRL in the 
classroom. 

Data will be collected daily for a four-week period. 

Methodology: 

Research will be conducted over a four-week period. All documentation will be 
turned to Beth Bingman in January 2000. 

We will use Bloom’s Taxonomy and the action words listed on the attached chart to 
describe indicators of TRL. The process will be: 

1 . Each teacher will give students a pre-survey on TRL. 

2. Teachers will select a model story that reflects the standard and use RWD to 
present the story. 

3. The teacher will teach a lesson on TRL and provide examples to students. 

4. Teachers will work with students to get buy-in on this four-week project. 

5. Students will be ask to keep notes, write in a journal, or tape record events that 
happen in their daily lives and report those back to the teacher each day. They 
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will be given 3-5 skill wheels to take home each day. When they report for 
school in the mornings or evenings they will discuss events with their teacher, 
turn in writings, and submit any evidence they choose and skill wheels that 
indicate skills they used. 

6. Teachers will collect information daily. They will analyze and describe the 
learning and link it back to TRL and/or other skills. This means teachers need to 
listen and discuss events with students, document findings, and collect products. 
Start out early recording observations and analysis in journals. 
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NC9ALL 



National Center for tke St udy of Adult Learning and Literacy 



NCSALL’s Mission 

The National Center for the Study of Adult Learning and Literacy (NCSALL) provides information 
used to improve practice in programs that offer adult basic education, English for speakers of other 
languages, and adult secondary education. In pursuit of this goal, NCSALL has undertaken research 
in four areas: learner motivation, classroom practice and the teaching/leaming interaction, staff 
development, and assessment. 

NCSALL conducts basic and applied research; builds partnerships between researchers and 
practitioners; disseminates research and best practices to practitioners, scholars, and policymakers; 
and works with the field of adult literacy education to develop a comprehensive research agenda. 

NCSALL is a partnership of the Harvard Graduate School of Education, World Education, 
Rutgers University, Portland State University in Oregon, and the Center for Literacy Studies at the 
University of Tennessee in Knoxville. NCSALL receives funding from the U.S. Department of 
Education’s Office of Educational Research and Improvement, National Institute for Postsecondary 
Education, Libraries, and Lifelong Learning; the Wallace-Reader’s Digest Funds; the National 
Institute for Literacy; and the Office of Vocational and Adult Education. 

NCSALL’s Dissemination Initiative 

NCSALL’s dissemination initiative focuses on ensuring that the research results reach practitioners, 
administrators, policymakers, and scholars of adult education through print, electronic, and face-to- 
face communication. NCSALL publishes research reports, occasional papers, research briefs, and 
teaching and training materials; the quarterly journal Focus on Basics', and The Annual Review of 
Adult Learning and Literacy, a scholarly review of major issues, current research, and best 
practices. 

For more information about NCSALL, to download free copies of NCSALL publications, 
or to purchase bound copies, please visit: 

http://ncsall.gse.harvard.edu 
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